Perceiving depth from texture and disparity cues: Evidence for a non-probabilistic account of cue integration.

Bayesian inference theories have been extensively used to model how the brain derives three-dimensional (3D) information from ambiguous visual input. In particular, the maximum likelihood estimation (MLE) model combines estimates from multiple depth cues according to their relative reliability to produce the most probable 3D interpretation. Here, we tested an alternative theory of cue integration, termed the intrinsic constraint (IC) theory, which postulates that the visual system derives the most stable, not most probable, interpretation of the visual input amid variations in viewing conditions. The vector sum model provides a normative approach for achieving this goal where individual cue estimates are components of a multidimensional vector whose norm determines the combined estimate. Individual cue estimates are not accurate but related to distal 3D properties through a deterministic mapping. In three experiments, we show that the IC theory can more adeptly account for 3D cue integration than MLE models. In Experiment 1, we show systematic biases in the perception of depth from texture and depth from binocular disparity. Critically, we demonstrate that the vector sum model predicts an increase in perceived depth when these cues are combined. In Experiment 2, we illustrate the IC theory radical reinterpretation of the just noticeable difference (JND) and test the related vector sum model prediction of the classic finding of smaller JNDs for combined-cue versus single-cue stimuli. In Experiment 3, we confirm the vector sum prediction that biases found in cue integration experiments cannot be attributed to flatness cues, as the MLE model predicts.

Memory, perceptual, and motor costs affect the strength of categorical encoding during motor learning of object properties.

Nearly all tasks of daily life involve skilled object manipulation, and successful manipulation requires knowledge of object dynamics. We recently developed a motor learning paradigm that reveals the categorical organization of motor memories of object dynamics. When participants repeatedly lift a constant-density "family" of cylindrical objects that vary in size, and then an outlier object with a greater density is interleaved into the sequence of lifts, they often fail to learn the weight of the outlier, persistently treating it as a family member despite repeated errors. Here we examine eight factors (Similarity, Cardinality, Frequency, History, Structure, Stochasticity, Persistence, and Time Pressure) that could influence the formation and retrieval of category representations in the outlier paradigm. In our web-based task, participants (N = 240) anticipated object weights by stretching a virtual spring attached to the top of each object. Using Bayesian t-tests, we analyze the relative impact of each manipulated factor on categorical encoding (strengthen, weaken, or no effect). Our results suggest that category representations of object weight are automatic, rigid, and linear and, as a consequence, the key determinant of whether an outlier is encoded as a member of the family is its discriminability from the family members.

Object weight can be rapidly predicted, with low cognitive load, by exploiting learned associations between the weights and locations of objects.

Weight prediction is critical for dexterous object manipulation. Previous work has focused on lifting objects presented in isolation and has examined how the visual appearance of an object is used to predict its weight. Here we tested the novel hypothesis that when interacting with multiple objects, as is common in everyday tasks, people exploit the locations of objects to directly predict their weights, bypassing slower and more demanding processing of visual properties to predict weight. Using a three-dimensional robotic and virtual reality system, we developed a task in which participants were presented with a set of objects. In each trial a randomly chosen object translated onto the participant's hand and they had to anticipate the object's weight by generating an equivalent upward force. Across conditions we could control whether the visual appearance and/or location of the objects were informative as to their weight. Using this task, and a set of analogous web-based experiments, we show that when location information was predictive of the objects' weights participants used this information to achieve faster prediction than observed when prediction is based on visual appearance. We suggest that by "caching" associations between locations and weights, the sensorimotor system can speed prediction while also lowering working memory demands involved in predicting weight from object visual properties.NEW & NOTEWORTHY We use a novel object support task using a three-dimensional robotic interface and virtual reality system to provide evidence that the locations of objects are used to predict their weights. Using location information, rather than the visual appearance of the objects, supports fast prediction, thereby avoiding processes that can be demanding on working memory.

Motor memories of object dynamics are categorically organized.

The ability to predict the dynamics of objects, linking applied force to motion, underlies our capacity to perform many of the tasks we carry out on a daily basis. Thus, a fundamental question is how the dynamics of the myriad objects we interact with are organized in memory. Using a custom-built three-dimensional robotic interface that allowed us to simulate objects of varying appearance and weight, we examined how participants learned the weights of sets of objects that they repeatedly lifted. We find strong support for the novel hypothesis that motor memories of object dynamics are organized categorically, in terms of families, based on covariation in their visual and mechanical properties. A striking prediction of this hypothesis, supported by our findings and not predicted by standard associative map models, is that outlier objects with weights that deviate from the family-predicted weight will never be learned despite causing repeated lifting errors.

Newtonian Predictions Are Integrated With Sensory Information in 3D Motion Perception.

Because the motions of everyday objects obey Newtonian mechanics, perhaps these laws or approximations thereof are internalized by the brain to facilitate motion perception. Shepard's seminal investigations of this hypothesis demonstrated that the visual system fills in missing information in a manner consistent with kinematic constraints. Here, we show that perception relies on internalized regularities not only when filling in missing information but also when available motion information is inconsistent with the expected outcome of a physical event. When healthy adult participants (Ns = 11, 11, 12, respectively, in Experiments 1, 2, and 3) viewed 3D billiard-ball collisions demonstrating varying degrees of consistency with Newtonian mechanics, their perceptual judgments of postcollision trajectories were biased toward the Newtonian outcome. These results were consistent with a maximum-likelihood model of sensory integration in which perceived target motion following a collision is a reliability-weighted average of a sensory estimate and an internal prediction consistent with Newtonian mechanics.

Persistent grasping errors produce depth cue reweighting in perception.

When a grasped object is larger or smaller than expected, haptic feedback automatically recalibrates motor planning. Intriguingly, haptic feedback can also affect 3D shape perception through a process called depth cue reweighting. Although signatures of cue reweighting also appear in motor behavior, it is unclear whether this motor reweighting is the result of upstream perceptual reweighting, or a separate process. We propose that perceptual reweighting is directly related to motor control; in particular, that it is caused by persistent, systematic movement errors that cannot be resolved by motor recalibration alone. In Experiment 1, we inversely varied texture and stereo cues to create a set of depth-metamer objects: when texture specified a deep object, stereo specified a shallow object, and vice versa, such that all objects appeared equally deep. The stereo-texture pairings that produced this perceptual metamerism were determined for each participant in a matching task (Pre-test). Next, participants repeatedly grasped these depth metamers, receiving haptic feedback that was positively correlated with one cue and negatively correlated with the other, resulting in persistent movement errors. Finally, participants repeated the perceptual matching task (Post-test). In the condition where haptic feedback reinforced the texture cue, perceptual changes were correlated with changes in grasping performance across individuals, demonstrating a link between perceptual reweighting and improved motor control. Experiment 2 showed that cue reweighting does not occur when movement errors are rapidly corrected by standard motor adaptation. These findings suggest a mutual dependency between perception and action, with perception directly guiding action, and actions producing error signals that drive motor and perceptual learning.

Sensorimotor adaptation and cue reweighting compensate for distorted 3D shape information, accounting for paradoxical perception-action dissociations.

Visually guided movements can show surprising accuracy even when the perceived three-dimensional (3D) shape of the target is distorted. One explanation of this paradox is that an evolutionarily specialized "vision-for-action" system provides accurate shape estimates by relying selectively on stereo information and ignoring less reliable sources of shape information like texture and shading. However, the key support for this hypothesis has come from studies that analyze average behavior across many visuomotor interactions where available sensory feedback reinforces stereo information. The present study, which carefully accounts for the effects of feedback, shows that visuomotor interactions with slanted surfaces are actually planned using the same cue-combination function as slant perception and that apparent dissociations can arise due to two distinct supervised learning processes: sensorimotor adaptation and cue reweighting. In two experiments, we show that when a distorted slant cue biases perception (e.g., surfaces appear flattened by a fixed amount), sensorimotor adaptation rapidly adjusts the planned grip orientation to compensate for this constant error. However, when the distorted slant cue is unreliable, leading to variable errors across a set of objects (i.e., some slants are overestimated, others underestimated), then relative cue weights are gradually adjusted to reduce the misleading effect of the unreliable cue, consistent with previous perceptual studies of cue reweighting. The speed and flexibility of these two forms of learning provide an alternative explanation of why perception and action are sometimes found to be dissociated in experiments where some 3D shape cues are consistent with sensory feedback while others are faulty.NEW & NOTEWORTHY When interacting with three-dimensional (3D) objects, sensory feedback is available that could improve future performance via supervised learning. Here we confirm that natural visuomotor interactions lead to sensorimotor adaptation and cue reweighting, two distinct learning processes uniquely suited to resolve errors caused by biased and noisy 3D shape cues. These findings explain why perception and action are often found to be dissociated in experiments where some cues are consistent with sensory feedback while others are faulty.

Depth cue reweighting requires altered correlations with haptic feedback.

Depth cue reweighting is a feedback-driven learning process that modifies the relative influences of different sources of three-dimensional shape information in perceptual judgments and or motor planning. In this study, we investigated the mechanism supporting reweighting of stereo and texture information by manipulating the haptic feedback obtained during a series of grasping movements. At the end of each grasp, the fingers closed down on a physical object that was consistent with one of the two cues, depending on the condition. Previous studies have shown that this style of visuomotor training leads to cue reweighting for perceptual judgments, but the time course has never been documented for a single training session, and many questions remain regarding the underlying mechanism, such as the pattern of feedback signals required to drive reweighting. We address these issues in two experiments, finding short-term changes in the motor response consistent with cue reweighting: the slope of the grip aperture with respect to the reliable cue increased, whereas the slope with respect to the unreliable cue decreased. Critically, Experiment 2 shows that slope changes do not occur when one of the cues is rendered with a constant bias; the grip aperture simply becomes uniformly larger or smaller. Our findings support a model of cue reweighting driven by altered correlations between haptic feedback and individual cues, rather than simple mismatches, which can be resolved by other mechanisms such as sensorimotor adaptation or cue recalibration.

Transfer of adaptation reveals shared mechanism in grasping and manual estimation.

An influential idea in cognitive neuroscience is that perception and action are highly separable brain functions, implemented in distinct neural systems. In particular, this theory predicts that the functional distinction between grasping, a skilled action, and manual estimation, a type of perceptual report, should be mirrored by a split between their respective control systems. This idea has received support from a variety of dissociations, yet many of these findings have been criticized for failing to pinpoint the source of the dissociation. In this study, we devised a novel approach to this question, first targeting specific grasp control mechanisms through visuomotor adaptation, then testing whether adapted mechanisms were also involved in manual estimation - a response widely characterized as perceptual in function. Participants grasped objects in virtual reality that could appear larger or smaller than the actual physical sizes felt at the end of each grasp. After brief exposure to a size perturbation, manual estimates were biased in the same direction as the maximum grip apertures of grasping movements, indicating that the adapted mechanism is active in both tasks, regardless of the perception-action distinction. Additional experiments showed that the transfer effect generalizes broadly over space (Exp. 1B) and does not appear to arise from a change in visual perception (Exp. 2). We discuss two adaptable mechanisms that could have mediated the observed effect: (a) an afferent proprioceptive mechanism for sensing grip shape; and (b) an efferent visuomotor transformation of size information into a grip-shaping motor command.

Does visuomotor adaptation contribute to illusion-resistant grasping?

Do illusory distortions of perceived object size influence how wide the hand is opened during a grasping movement? Many studies on this question have reported illusion-resistant grasping, but this finding has been contradicted by other studies showing that grasping movements and perceptual judgments are equally susceptible. One largely unexplored explanation for these contradictions is that illusion effects on grasping can be reduced with repeated movements. Using a visuomotor adaptation paradigm, we investigated whether an adaptation model could predict the time course of Ponzo illusion effects on grasping. Participants performed a series of trials in which they viewed a thin wooden target, manually reported an estimate of the target's length, then reached to grasp the target. Manual size estimates (MSEs) were clearly biased by the illusion, but maximum grip apertures (MGAs) of grasping movements were consistently accurate. Illusion-resistant MGAs were observed immediately upon presentation of the illusion, so there was no decrement in susceptibility for the adaptation model to explain. To determine whether online corrections based on visual feedback could have produced illusion-resistant MGAs, we performed an exploratory post hoc analysis of movement trajectories. Early portions of the illusion effect profile evolved as if they were biased by the illusion to the same magnitude as the perceptual responses (MSEs), but this bias was attenuated prior to the MGA. Overall, this preregistered study demonstrated that visuomotor adaptation of grasping is not the primary source of illusion resistance in closed-loop grasping.

Error correction and spatial generalization in human grasp control.

The visual processes that support grasp planning are often studied by analyzing averaged kinematics of repeated movements, as in the literature on grasping and visual illusions. However, by recalibrating visuomotor mappings, the sensorimotor system can adjust motor outputs without changing visual processing, which complicates the interpretation of averaged behavior. We developed a dynamic model of grasp planning and adaptation that can explain why some studies find decrements in illusion effects on grasping while others do not. In two experiments, we tested grasping in a standard three-phase adaptation paradigm and analyzed adaptation aftereffects on the maximum grip aperture as well as the error correction parameters estimated by our model. Experiment 1 demonstrated that the model accounts for adaptive responses to positive and negative visual size perturbations. Experiment 2 supported the novel hypothesis that visuomotor mappings for grasp planning can compensate for opposing size perturbations when these perturbations are experienced in separate regions of space. Our findings serve to illustrate how the surprising flexibility of grasp adaptation can hide (especially in session-wise averages) the true effects of visual perturbations on the visual processes that drive grasp planning.

Adaptation effects in grasping the Müller-Lyer illusion.

Recent results have shown that effects of pictorial illusions in grasping may decrease over the course of an experiment. This can be explained as an effect of sensorimotor learning if we consider a pictorial size illusion as simply a perturbation of visually perceived size. However, some studies have reported very constant illusion effects over trials. In the present paper, we apply an error-correction model of adaptation to experimental data of N=40 participants grasping the Müller-Lyer illusion. Specifically, participants grasped targets embedded in incremental and decremental Müller-Lyer illusion displays in (1) the same block in pseudo-randomised order, and (2) separate blocks of only one type of illusion each. Consistent with predictions of our model, we found an effect of interference between the two types when they were presented intermixed, explaining why adaptation rates may vary depending on the experimental design. We also systematically varied the number of object sizes per block, which turned out to have no effect on the rate of adaptation. This was also in accordance with our model. We discuss implications for the illusion literature, and lay out how error-correction models can explain perception-action dissociations in some, but not all grasping-of-illusion paradigms in a parsimonious and plausible way, without assuming different illusion effects.